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Introduction: 


The  Society  for  Simulation  in  Healthcare  convened  the  second  Simulation  Education 
Summit  meeting  in  October  2007  in  Chicago,  Illinois.  The  purpose  of  the  Summit  was 
to  bring  together  leaders  of  public,  private,  and  government  organizations,  associations, 
and  agencies  involved  in  healthcare  education  for  a  focused  discussion  of  standards  for 
simulation-based  applications.  Sixty-eight  participants  representing  36  organizations 
discussed  in  structured  small  and  large  groups  the  criteria  needed  for  various  training 
and  assessment  applications  using  simulation.  While  consensus  was  reached  for  many 
topics,  there  were  also  areas  that  required  further  thought  and  dialogue 

Body: 

Sixty-eight  individuals  representing  36  organizations  participated  in  the  daylong  meeting  (Appendix  A). 
Many  of  the  invited  attendees  had  also  attended  the  2006  Summit.  Everyone  involved  had  a  broad 
spectrum  of  simulation  education  experience  and  came  from  organizations  with  an  interest  in  simulation, 
including  specialty  societies,  regulatory  bodies,  and  industry.  SSH  board  members  were  also  present. 

Dan  Raemer,  past  president  of  SSH  and  chair  of  the  Summit  planning  committee,  started  the  meeting 
with  a  few  videos  about  the  direction  and  use  of  simulation.  This  was  followed  by  an  overview  of  the 
meeting  objectives  and  a  short  simulation  taxonomy  ice-breaker  exercise  using  an  audience  response 
system.  Participants  responded  to  the  degree  of  agreement  with  the  provided  definition  of  simulation 
terms  (Appendix  B).  Those  who  disagree  or  strongly  disagree  were  asked  to  write  down  their  comments. 

A  professional  facilitator  (Jay  Vogt,  Peoplesworth,  Concord,  MA)  led  a  series  of  working  sessions  where 
different  groups  addressed  the  following  areas  related  to  simulation: 

•  Formative  and  Summative  Assessment  of  Trainees 

•  Formative  and  Summative  Assessment  of  Practitioners 

•  Continuing  Medical/Healthcare  Education 

•  Formative  and  Summative  Assessment  of  New  Procedures 

•  Substitute  for  Clinical  Encounter 

•  Rehabilitation  and  Remediation  of  Healthcare  Practitioners 

•  Credentialing  of  Practitioners 

•  Patient  Safety 

For  each  topic,  participants  were  asked  to  discuss  acceptable  standards  regarding  the  following: 

•  Simulators  -  mode/type  (e.g.  standardized  patients,  screen-based,  task  trainers,  mannequin- 
based,  virtual  reality,  etc.),  equipment  fidelity  and  validity  issues 

•  Instructors  -  qualifications,  need  for  formal  certification 

•  Metrics  -  basic  competencies  that  simulation  is  expected  to  measure 

•  Evaluation  -  valid  and  reliable  subjective/objective  assessment  tools 

•  Ratification  -  approval  process  for  standards  and  verification  of  compliance 

•  Time  -  requirements  related  to  duration  and  frequency  of  simulation  training 

•  Other  elements  -  differences  between  standards  for  technical  vs.  cognitive  or  critical  thinking 
skills,  training  vs.  assessment,  formative  vs.  summative;  other  considerations 

Participants  were  also  asked  to  discuss  the  specific  training  and  assessment  challenges  facing  their 
organization,  the  assessment  methods  that  their  organization  would  most  likely  accept  and  the  criteria 
that  would  be  required  or  recommended  for  implementation  of  simulation  activities  and  standards. 
Participants  first  worked  in  small  groups  of  7-8  at  an  assigned  table.  Two  tables  were  given  the  same 
topic  to  discuss.  After  an  hour  of  discussion,  the  two  tables  with  the  same  topic  shared  their  ideas  to 
reach  a  consensus.  A  combined  report  was  given  to  the  entire  Summit  group.  These  reports  were 
documented  by  each  group  and  collated  by  the  facilitator  (JV)  who  outlined  all  the  notes  on  a  34-page 
report,  which  this  executive  summary  report  was  based  on.  The  final  summary  report  was  then  circulated 
to  all  participants  of  the  Summit  for  feedback. 
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The  Meeting  report  will  be  published  in  Vol  3  Issue  3  of  the  Society’s  Journal, 
Simulation  in  Healthcare  and  are  also  available  on-line. 


Key  Research  Accomplishments: 

After  six  hours  of  brainstorming  and  discussions,  an  important  consensus  was  established:  more  of  these 
collaborative  meetings  are  needed  and  more  time  was  required  to  fully  tackle  the  issues  at  hand.  A  single 
day  of  meeting  was  insufficient  to  deliver  a  comprehensive  standards  guideline.  However,  despite  the 
challenging  feat,  there  were  some  preliminary  findings  that  were  established  and  could  serve  as  the 
framework  for  future  working  meetings. 

To  begin,  the  taxonomy  exercise  showed  that  in  general,  most  people  tended  to  agree  with  the  provided 
definitions  (average  response  3.6-4. 1  for  all  questions,  on  a  Likert  scale  with  1=strongly  disagree, 
5=strongly  disagree).  Only  a  couple  of  individuals  submitted  comments;  one  noted  that  “certification”  is  a 
process  for  qualifying  individuals  and  “accreditation”  was  the  more  appropriate  word  for  the  process  for 
approving  simulation  programs.  Additional  definitions,  such  as  standards  vs.  guidelines  vs.  template  vs. 
recommendations  were  desired.  Other  semantic  questions  were  raised  in  regards  to  credentialing  vs. 
competency  certification. 

During  the  large  group  reports,  it  was  clear  that  there  was  consensus  regarding  specific  standards  that 
should  be  considered  regardless  of  the  simulation  application.  The  overarching  theme  was  that 
everything  should  be  driven  by  the  educational  objectives.  Moreover,  a  number  of  other  unifying  threads 
cut  across  all  topics  discussed. 


Reportable  Outcomes: 

The  increasing  multidisciplinary  membership  of  Society  for  Simulation  in  Healthcare  has  generated  a 
demand  for  consistent  and  validated  curricula  and  assessment  tools  for  the  expanding  applications  using 
simulation,  as  evident  by  the  questions  posed  on  the  SSH  listserv.  The  mission  of  SSH  is  to  facilitate 
excellent  multi-specialty  healthcare  education,  practice,  and  research  through  simulation.  Therefore,  to 
better  represent  our  multidisciplinary  simulation  society,  we  need  to  consolidate  the  needs  and  desires  of 
different  agencies  and  healthcare  specialties  in  order  to  establish  and  implement  a  unified  set  of 
educational  standards  for  simulation  applications. 

This  second  Simulation  Education  Summit  is  a  testament  to  the  wonders  of  collaborative  efforts.  As  a 
result  of  bringing  together  the  key  stakeholders  in  simulation  education,  this  preliminary  outline  of 
considerations  for  simulation-based  training  and  assessment  standards  has  been  drafted.  There  is  no 
doubt  that  much  more  work  is  needed  before  a  more  comprehensive  set  of  standards  or  a  guidelines 
document  can  be  distributed.  However,  this  report  serves  as  a  starting  point  to  address  the  same 
questions  that  every  organization  faces.  The  fact  that  nearly  all  Summit  participants  indicated  that  we 
should  continue  to  meet  in  this  collaborative  forum  and  not  wait  a  year  to  reconvene  is  an  indication  of  the 
strong  desire  to  work  together  toward  a  common  goal.  At  the  conclusion  of  the  meeting,  the  group  agreed 
on  an  action  plan. 

Conclusions: 

•  The  Society  for  Simulation  in  Healthcare  will  form  a  coalition  of  societies  interested  in  simulation 
to  serve  as  the  “think  tank”  in  resolving  simulation-based  issues. 

•  After  further  discussion  and  consensus,  SSH  will  convene  an  ad  hoc  development  group  to 
create  the  standards  documents  and  guidelines  for  review  and  dissemination. 

•  The  next  meeting  should  include  definitions  of  standards  and  contextual  framing  to  facilitate 
better  discussions. 

•  SSH  should  act  as  a  clearinghouse  to  survey  practices  within  organizations. 

•  Along  with  input  from  specialty  societies  and  accrediting  agencies,  SSH  will  continue  to  lead 
these  efforts  in  consolidating  common  simulation  needs  and  issues. 
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American  College  of  Chest  Physicians,  Northwestern  Feinberg  School  of  Medicine,  Center  for  Medical 

Simulation 


Abstract: 

The  Society  for  Simulation  in  Healthcare  convened  the  second  Simulation  Education  Summit  meeting  in 
October  2007  in  Chicago,  Illinois.  The  purpose  of  the  Summit  was  to  bring  together  leaders  of  public, 
private,  and  government  organizations,  associations,  and  agencies  involved  in  healthcare  education  for  a 
focused  discussion  of  standards  for  simulation-based  applications.  Sixty-eight  participants  representing 
36  organizations  discussed  in  structured  small  and  large  groups  the  criteria  needed  for  various  training 
and  assessment  applications  using  simulation.  While  consensus  was  reached  for  many  topics,  there  were 
also  areas  that  required  further  thought  and  dialogue.  This  paper  is  a  summary  of  the  results  of  these 
discussions  along  with  a  preliminary  draft  of  a  guideline  for  simulation-based  education. 

Introduction: 

The  Society  for  Simulation  in  Healthcare  (SSH)  organized  its  second  Simulation  Education  Summit 
meeting  on  October  26,  2007  in  Chicago,  Illinois.  The  2007  Summit  was  initiated  in  response  to  the  first 
Simulation  Summit  held  at  the  same  site  in  November  2006, 1  which  resulted  in  a  consensus  call  for  SSH 
to  serve  as  the  consolidator  for  simulation  activities.  The  goal  of  the  first  Summit  was  to  advance 
simulation  into  mainstream  healthcare  education  and  as  a  product  of  the  meeting,  a  list  of  important 
topics  and  priorities  for  future  meetings  was  generated.  The  primary  goal  for  the  2007  Summit  meeting 
was  to  create  a  working  draft  of  standards  for  various  kinds  of  learning  and  assessment  when  simulation 
is  used  as  a  substantial  part  of  the  process. 

Simulation  standards  and  guidelines  were  chosen  for  the  2007  focus  because  they  were  among  the  first 
tier  topics  discussed  at  the  2006  Summit.  Along  with  establishing  standards,  competency  metrics  and 
assessment  were  also  on  the  agenda.  The  desired  meeting  outcomes  included  building  consensus  for 
important  simulation  terms,  determining  the  levels  of  assessment  and  acceptable  methods  for 
assessment  using  simulation  and  developing  an  action  plan  for  next  steps. 

Method: 

Sixty-eight  individuals  representing  36  organizations  participated  in  the  daylong  meeting  (Appendix  A). 
Many  of  the  invited  attendees  had  also  attended  the  2006  Summit.  Everyone  involved  had  a  broad 
spectrum  of  simulation  education  experience  and  came  from  organizations  with  an  interest  in  simulation, 
including  specialty  societies,  regulatory  bodies,  and  industry.  SSH  board  members  were  also  present. 

Dan  Raemer,  past  president  of  SSH  and  chair  of  the  Summit  planning  committee,  started  the  meeting 
with  a  few  videos  about  the  direction  and  use  of  simulation.  This  was  followed  by  an  overview  of  the 
meeting  objectives  and  a  short  simulation  taxonomy  ice-breaker  exercise  using  an  audience  response 
system.  Participants  responded  to  the  degree  of  agreement  with  the  provided  definition  of  simulation 
terms  (Appendix  B).  Those  who  disagree  or  strongly  disagree  were  asked  to  write  down  their  comments. 

A  professional  facilitator  (Jay  Vogt,  Peoplesworth,  Concord,  MA)  led  a  series  of  working  sessions  where 
different  groups  addressed  the  following  areas  related  to  simulation: 

•  Formative  and  Summative  Assessment  of  Trainees 

•  Formative  and  Summative  Assessment  of  Practitioners 

•  Continuing  Medical/Healthcare  Education 

•  Formative  and  Summative  Assessment  of  New  Procedures 
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•  Substitute  for  Clinical  Encounter 

•  Rehabilitation  and  Remediation  of  Healthcare  Practitioners 

•  Credentialing  of  Practitioners 

•  Patient  Safety 

For  each  topic,  participants  were  asked  to  discuss  acceptable  standards  regarding  the  following: 

•  Simulators  -  mode/type  (e.g.  standardized  patients,  screen-based,  task  trainers,  mannequin- 
based,  virtual  reality,  etc.),  equipment  fidelity  and  validity  issues 

•  Instructors  -  qualifications,  need  for  formal  certification 

•  Metrics  -  basic  competencies  that  simulation  is  expected  to  measure 

•  Evaluation  -  valid  and  reliable  subjective/objective  assessment  tools 

•  Ratification  -  approval  process  for  standards  and  verification  of  compliance 

•  Time  -  requirements  related  to  duration  and  frequency  of  simulation  training 

•  Other  elements  -  differences  between  standards  for  technical  vs.  cognitive  or  critical  thinking 
skills,  training  vs.  assessment,  formative  vs.  summative;  other  considerations 

Participants  were  also  asked  to  discuss  the  specific  training  and  assessment  challenges  facing  their 
organization,  the  assessment  methods  that  their  organization  would  most  likely  accept  and  the  criteria 
that  would  be  required  or  recommended  for  implementation  of  simulation  activities  and  standards. 
Participants  first  worked  in  small  groups  of  7-8  at  an  assigned  table.  Two  tables  were  given  the  same 
topic  to  discuss.  After  an  hour  of  discussion,  the  two  tables  with  the  same  topic  shared  their  ideas  to 
reach  a  consensus.  A  combined  report  was  given  to  the  entire  Summit  group.  These  reports  were 
documented  by  each  group  and  collated  by  the  facilitator  (JV)  who  outlined  all  the  notes  on  a  34-page 
report,  which  this  executive  summary  report  was  based  on.  The  final  summary  report  was  then  circulated 
to  all  participants  of  the  Summit  for  feedback. 

Results 

After  six  hours  of  brainstorming  and  discussions,  an  important  consensus  was  established:  more  of  these 
collaborative  meetings  are  needed  and  more  time  was  required  to  fully  tackle  the  issues  at  hand.  A  single 
day  of  meeting  was  insufficient  to  deliver  a  comprehensive  standards  guideline.  However,  despite  the 
challenging  feat,  there  were  some  preliminary  findings  that  were  established  and  could  serve  as  the 
framework  for  future  working  meetings. 

To  begin,  the  taxonomy  exercise  showed  that  in  general,  most  people  tended  to  agree  with  the  provided 
definitions  (average  response  3.6-4. 1  for  all  questions,  on  a  Likert  scale  with  1=strongly  disagree, 
5=strongly  disagree).  Only  a  couple  of  individuals  submitted  comments;  one  noted  that  “certification”  is  a 
process  for  qualifying  individuals  and  “accreditation”  was  the  more  appropriate  word  for  the  process  for 
approving  simulation  programs.  Additional  definitions,  such  as  standards  vs.  guidelines  vs.  template  vs. 
recommendations  were  desired.  Other  semantic  questions  were  raised  in  regards  to  credentialing  vs. 
competency  certification. 

Unifying  Themes 

During  the  large  group  reports,  it  was  clear  that  there  was  consensus  regarding  specific  standards  that 
should  be  considered  regardless  of  the  simulation  application.  The  overarching  theme  was  that 
everything  should  be  driven  by  the  educational  objectives.  Moreover,  a  number  of  other  unifying  threads 
cut  across  all  topics  discussed. 


Simulators 

All  conceded  that  if  an  organization  is  using  clinical  simulation  activity,  it  must  specify  specific  learning 
objectives  and  identify  appropriate  simulation  activity  to  match  the  objectives.  The  simulator(s)  or 
simulation  exercise  should  have  at  minimum  construct  validity  (i.e.  the  simulator  equipment  does  what  it  is 
supposed  to  do).  A  combination  of  various  types  of  simulations  were  encouraged.  Simulations  used  for 
formative  evaluation  should  be  the  same  as  those  used  for  summative  assessment  for  validation 
purposes,  although  for  the  purpose  of  practice,  lower-end  models  may  suffice.  High  stakes  activities  will 
require  higher  standards  (e.g.  validated  simulators)  and  a  greater  degree  of  standardization. 

Instructors 

It  was  uniformly  agreed  upon  that  instructors  must  have  appropriate  simulation  training  and  be  able  to 
demonstrate  basic  competency  with  the  simulation  system  being  used.  Also,  they  must  have  validated 
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content  knowledge  as  specified  by  the  educational  objectives.  What  constitutes  “appropriate”  simulation 
training  was  not  fully  discussed.  Participants  also  did  not  reach  an  agreement  regarding  whether  or  not 
instructors  will  need  to  be  formally  trained  via  a  recognized  simulation  training  program,  or  what  would 
qualify  as  an  adequate  training  center  to  certify  instructors.  Analogies  were  made  to  successful  programs 
such  as  ACLS  and  ATLS  where  instructors  must  be  certified  and  their  certifications  renewed  every  few 
years. 


Metrics  &  Evaluation 

It  was  generally  agreed  that  metrics  and  evaluation  tools  should  be  addressed  together.  Metrics  are 
standards  of  measurement  and  evaluation  tools  are  the  specific  means  of  determining  whether  the 
metrics  are  met.  The  metrics  should  be  competency-based  and  driven  by  the  learning  objectives  and  the 
learner  needs.  The  metrics  should  include  cognitive,  psychomotor,  and  affective  aspects  of  learning. 
These  could  be  broken  down  into  specific  knowledge,  skills,  attitudes  and  behavioral  competencies. 

A  number  of  evaluation  modalities  were  endorsed  including  checklists,  360-degree  feedback,  video, 
standardized  patient  review/feedback,  self-review/feedback,  test  scores,  simulator  feedback,  expert 
review/feedback,  and  demonstrations  of  transfer  of  knowledge  to  patients  or  peers.  It  is  important  to  note 
that  simulation  is  recognized  as  only  one  of  many  tools  within  an  individual’s  performance  portfolio. 
Validated  evaluation  tools  are  clearly  desirable  and  should  be  a  goal  but  may  not  be  available  for  all 
metrics  at  this  time. 


Ratification 

Most  participants  felt  that  it  is  currently  too  soon  to  set  a  process  for  approving  these  simulation 
standards.  However,  in  some  areas  certain  considerations  were  proposed.  Input  from  accrediting 
agencies  is  clearly  needed. 


Time 

There  was  general  consensus  that  an  organization  must  specify  the  appropriate  duration  and  frequency 
of  any  simulation  activity  needed  to  achieve  their  learning  objectives.  More  time  in  practice  is  desired. 

Other  Elements 

Issues  such  as  who  should  certify  instructors  or  approve  simulation  standards,  cost  considerations,  and 
legal  challenges  were  discussed  but  no  consensus  was  reached  at  this  time. 

Formative  and  Summative  Assessment  of  Trainees 

Trainees  are  defined  as  residents,  medical  students,  nursing  students,  dental  students,  allied  health 
students,  hospital  service  students,  “family”  students,  preparaprofessionals  or  teams  of  students. 

In  addition  to  the  common  unifying  themes,  simulation  use  at  the  trainee  level  should  also  consider  the 
issues  of  integration  of  simulation  into  the  existing  curricula.  A  participant’s  exposure  to  simulation  is 
driven  by  simulator  availability,  cost,  learner  needs,  learning  objectives,  and  encounter  history.  The  time 
spent  in  training  is  driven  by  availability  of  time  off  from  clinical  duties,  technology,  task  complexity  and 
student  capabilities. 

Formative  and  Summative  Assessment  of  Practitioners 

Practitioners  include  any  licensed  professional  who  provides  healthcare  directly  to  patients.  The 
environment  they  practice  is  any  place  where  there  is  direct  patient  care,  whether  at  academic  sites, 
private/non-academic  areas,  clinics,  or  community  health  centers. 

The  overall  opinion  was  that  simulator  use  must  be  standardized  for  formative  and  summative 
assessment  of  practitioners.  The  extent  of  standardization  remains  nebulous.  Different  modalities  may  be 
used  including  standardized  patients,  unannounced  simulated  patients,  virtual  reality,  low  and  high  fidelity 
simulators,  etc.  Homemade  simulators  that  cannot  be  validated  or  standardized  should  not  be  used  for 
assessment.  The  simulator  mode  and  characteristics  should  be  flexible  and  a  multi-modal  approach 
should  be  considered.  Simulators,  instructors,  metrics,  evaluation  and  exposure  should  support  the 
objectives:  objectives  ->  simulator  ->  evaluation  ->  objectives. 

A  formally  certified  evaluator  who  has  documented  content  expertise  in  the  specialty  area  and 
demonstrated  simulation/educational  skills  is  needed  to  perform  high  stakes  summative  assessment. 
Clinical  relevance  confers  credibility;  so  at  least  one  instructor/assessor  needs  to  be  in  the  specialty. 
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Metrics  need  to  be  specialty-specific  but  based  on  core  competencies  prescribed  by  the  profession. 
Performance  level  of  proficiency  needs  to  be  defined  and  used  as  a  goal  for  both  formative  and 
summative  evaluations.  Evaluation  tools  need  to  be  tested  and  validated  to  an  accepted  standard  by  the 
specialty  group.  Validation  should  be  evidence  based,  scientifically  defensible  in  a  legal  challenge,  and 
should  be  published  in  a  peer-reviewed  journal.  Examples  of  validated  tools  are  shown  in  Table  1. 

Time  is  limited  for  practitioners  to  train  with  simulation,  so  the  recommended  interval  should  be  3-5  years 
and  may  be  determined  by  the  specialty.  However,  formative  assessments  should  be  conducted  annually, 
and  may  not  necessarily  involve  simulation.  Certification  and  competency  validation  should  be  defined.  In 
accordance  with  board  certification  requirements,  certification  with  simulation  training  should  be  every  ten 
years. 

In  regards  to  ratification,  standards  for  use  of  simulation  for  formative  and  summative  assessment  of 
practitioners  should  be  reviewed  and  approved  by  a  committee  comprised  of  members  of  the  profession 
as  well  as  educators  and  experts  from  outside  the  profession  who  may  be  affected  by  the  standards.  An 
ongoing  process  for  feedback  along  with  continuous  research  is  desired. 

Continuing  Medical/Healthcare  Education  (CME/CE) 

In  general,  continuing  medical/healthcare  education  is  about  learning,  not  testing.  Thus,  CME/CE  must 
not  be  mistaken  for  certification  and  should  be  seen  as  a  mechanism  for  skills/knowledge  training  rather 
than  formative  assessments  or  summative  assessments,  as  these  standards  are  different.  The  mode  of 
simulation  should  include  all  modes  appropriate  for  the  CME/CE  population  and  tailored  to  the  goals  and 
objectives  of  the  CME/CE  curriculum.  Validation  may  be  needed  for  the  transfer  of  knowledge/skills  for 
obtaining  the  achievable  outcomes,  orfordefensibility  in  court. 

In  regards  to  metrics,  both  intracourse  (CME/CE  activity)  versus  extracourse  metrics  (implementation, 
change  in  practice  or  patient  outcome)  should  be  considered.  Evaluation  tools  should  specifically  define 
what  is  correct  and  incorrect  to  provide  feedback  for  improvement.  Duration  and  frequency  of  simulation 
activity  will  depend  on  skills  taught  and  should  be  proficiency  based.  High  stakes,  low  frequency  events 
need  more  frequent  training  than  common  skills.  The  qualification  of  the  instructors  should  follow  the 
same  principles  described  in  the  standards  for  formative  and  summative  assessment  of  practitioners. 

CME  providers  and  accrediting  agencies  will  be  responsible  for  ensuring  that  these  standards,  once 
approved  through  boards  of  relevant  societies,  are  enforced  by  the  various  institutions  that  provide 
simulation-based  CME.  Examples  are  stamps  of  approval  from  national  organizations,  such  as  the 
American  Society  of  Anesthesiologists  simulation  endorsement  program  and  the  American  College  of 
Surgeons  accreditation  program  for  those  centers  that  meet  that  specialty’s  CME  criteria. 

Formative  and  Summative  Assessment  of  New  Procedures 

The  mode  and  the  type  of  simulation  should  support  the  objectives  and  the  specific  step-wise  tasks 
involved  in  the  procedures.  Differentiation  of  manual  procedures  (individual-based)  vs.  hospital 
procedures  (team-based)  should  be  specified.  Any  simulation  mode  could  be  used  including  combination 
types.  The  standards  would  be  different  for  formative  vs.  summative  assessment  and  for  technical  vs. 
cognitive  skills.  High-risk  procedures  will  require  more  than  one  level  of  simulation.  Validation  is 
essential,  particularly  for  summative  assessment  and  to  determine  whether  simulators  that  act  as  their 
own  instructors  are  realistic,  valid  and  reliable.  However,  for  formative  purposes,  validation  needs  to  be 
balanced  against  cost  and  availability. 

The  instructors,  metrics,  evaluation  and  exposure  should  be  dictated  by  the  objectives  and  the  complexity 
of  the  procedure.  Instructors  should  be  formally  certified  unless  the  simulator  has  built  in  metrics.  For 
simulators  with  built  in  metrics,  the  role  of  the  instructor  may  be  to  evaluate  whether  or  not  someone  is 
cognitively  ready  to  learn  the  new  skill  or  to  teach  the  cognitive  and  behavioral  aspects  to  those  who  have 
mastered  the  manual  skills.  Successful  completion  of  simulation  training  should  be  followed  by  a  graded 
observation  of  a  real  case  performance.  A  Masters  in  Simulation  degree  program  was  proposed. 

Metrics  for  new  procedures  include  simple  and  complex  tasks,  psychomotor  and  cognitive  skills 
competencies.  Examples  include:  ability  to  adapt  to  non-standard  situations,  efficiency  completing 
simulation  (time,  accuracy,  success  rates),  adverse  events  and  managing  complications  (rate  and 
techniques  for  managing  complications),  decision  making  pathway,  success  in  performing  basic  tasks, 
correct  technique  and  correct  procedures,  perception  of  performance,  attitudes  towards 
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device/procedure,  communication,  dexterity  and  flexibility  in  tissue  handling,  judgment  and  timing  of 
events. 

Evaluations  should  be  both  objective  (checklist  of  tasks,  time,  sequence  of  events,  diagnostic  accuracy, 
correlation  to  clinical  experiences)  and  subjective  (judgment,  perceptions).  Training  time  requirements  are 
based  on  time  to  proficiency  (depends  on  learner  level)  with  a  required  minimum  number  of  practice 
sessions. 

Ratification  may  be  through  institution  for  compliance  with  standards,  FDA  for  device  qualifications,  or 
other  state/federal  or  payor  agencies  with  responsibilities  and  authorities  in  content  area.  An 
interdisciplinary  consensus  process  should  be  established  to  approve  these  standards. 

Other  issues  considered  for  new  procedures:  Do  learners  need  to  have  a  priori  knowledge  of  using  a 
simulator?  When  should  training/assessment  start?  Formative  standards  would  be  different  from 
summative  (higher  standards)  and  technical  vs.  cognitive  skills. 

Substitute  for  Clinical  Encounter 

Simulators  should  possess  the  appropriate  level  of  fidelity  to  mimic  real  life  patient  encounters  or 
experiences  and  events.  Organizations  using  clinical  simulation  must  specify  the  specific  learning 
objectives  (and  what  can  be  substituted)  and  identify  the  appropriate  simulation  activity  to  fulfill  those 
objectives.  A  strong  rationale  or  evidence  for  substituting  or  supplementing  clinical  activities  is  needed. 
The  instructors,  metrics,  evaluation  and  the  time  should  be  determined  based  on  the  objectives. 

Standards  are  the  same  for  technical  versus  cognitive  skills  training/assessment. 

Rehabilitation  and  Remediation  of  Healthcare  Practitioners 

The  first  issue  at  hand  is  to  identify  practitioners  who  are  in  need  of  remediation.  Current  remediation 
programs  are  geared  toward  private  substance  abuse  rehabilitation  or  ethical  conduct  remediation 
programs.  Working  in  conjunction  with  state  boards  or  other  accreditation  bodies  may  be  helpful  in 
addition  to  using  simulation  to  assess  competency  level.  Simulation  can  also  be  used  as  a  diagnostic  tool 
for  remediation  before  credentialing.  The  simulation  training  should  be  specific  to  the  deficiency.  An 
educational  specialist  is  needed  to  identify  learning  deficiencies  and  to  help  design  an  individualized 
educational  program  (IEP).  Human  feedback  is  required  even  if  the  simulator  has  built  in  performance 
indicators. 

A  sample  curricular  template  might  be: 

Diagnosis  of  need  for  rehabilitation/remediation  (through  simulation  and/or  from  state  boards)  ->  design 
IEP  ->  define  objectives  (knowledge,  skills  or  behavioral  goals)  ->  determine  appropriate  simulation  -> 
pre-intervention  assessment  ->  simulation  training  ->  post-intervention  evaluation  of  performance  ->  pass 
or  further  diagnosis. 

A  probationary  period  for  remediated  individuals  was  suggested,  i.e.  these  individuals  would  be  evaluated 
more  frequently  than  mainstream  practitioners.  However  legal  implications  should  be  considered. 
Ratification  should  be  from  credentialing  boards.  Private  agencies  or  academic  institutions  may  be  called 
upon  to  support  the  remediation  programs. 

Credentialing  of  Practitioners 

High  validity  and  standardization  are  mandatory  when  simulation  is  used  for  credentialing.  Standards 
should  be  driven  by  specialty  boards  rather  than  external  societies.  There  was  debate  in  regards  to 
“credentialing”  since  it  implies  not  only  high-stakes  assessment  of  adequate  skill  and  knowledge  but  also 
behavioral  characteristics  such  as  professionalism,  and  therefore  has  medicolegal  implications.  The 
debate  continued  in  regards  to  how  relevant  and  physical  is  simulation-based  assessment  for 
credentialing.  Credentialing  may  even  apply  to  learner  levels  if  simulation  is  to  be  used  for  high-stakes 
performance  assessment,  e.g.  advancement  of  medical  students  and  residents  to  the  next  year  of 
training. 

The  instructors,  metrics,  evaluation  and  time  required  for  credentialing  will  need  to  be  determined  by  the 
objective  and  standards  for  the  specialty.  Instructors  need  to  be  expert  judges  at  the  identified  deficiency 
and  must  be  required  to  take  specialized  training.  The  standards  should  be  set  by  national  organizations 
for  clinical  issues.  Validation  should  link  simulation  performance  to  patient  outcomes.  As  there  are  few 
validated  tools  at  this  point,  simulation  should  be  used  along  with  other  tools  for  credentialing,  not  by 
itself.  Multiple  tries  should  be  allowed  to  pass  simulation  credentialing.  Ratification  may  be  accomplished 
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through  Joint  Commission,  payers,  insurers,  Pay  for  Performance,  or  3rd  party  organizations,  but  it  is  still 
a  premature  concept  at  this  time. 

Some  issues  were  raised  about  the  implications  of  credentialing  on  macroeconomics,  e.g.  a  higher 
credentialing  bar  may  cause  clinicians  to  charge  more  money  or  be  less  likely  to  go  into  specific  fields, 
therefore  creating  the  risk  of  miscategorizing  good  people.  Moreover,  credentialing  may  be  irrelevant  to 
certain  groups,  as  local  oversight  of  nursing  and  allied  health  professional  skills  assessments  are  not 
considered  credentialing.  Finally,  there  is  the  issue  of  simulator  availability  for  practice. 

Patient  Safety 

Practitioner  competency  is  essential  for  patient  safety.  All  modes  of  simulation  are  appropriate;  again, 
they  must  be  linked  to  the  objectives.  Most  appropriate  applications  include:  high-risk  tasks  and  events, 
low-frequency  adverse  events,  teamwork  and  communication,  judgment  and  critical  thinking,  and 
procedural  skills.  The  instructors,  metrics,  evaluation  and  time  spent  on  the  activity  should  be  in 
accordance  with  the  objectives,  policies  and  procedures.  Instructors  could  be  content  experts  or 
simulation  experts.  Evaluation  tools  could  measure  errors  during  simulated  situations,  change  in 
organizational  cultures,  patient  satisfaction  and  perception  of  safety,  decrease  in  clinical  errors,  team 
performance  (through  mock  codes,  mock  patients)  and  individual  attitudes  and  skills. 

A  consensus  among  leaders  of  specialty  groups  is  needed  to  develop  a  process  for  assessment  as  well 
as  the  assessment  tools.  Ratification  may  be  done  through  mandated  enforcement  by  state  regulations  or 
Joint  Commission.  Simulation  experts  should  be  part  of  the  institutional  reviews.  It  is  also  important  to 
recognize  that  simulation  is  only  one  area  that  could  impact  patient  safety.  There  are  many  other  systems 
factors  and  human  factors  involved,  including  organizational  structure  and  policies,  and  society  guidelines 
and  standards.  Continuous  quality  improvement  measurements  are  needed  to  identify  and  assess  areas 
of  patient  safety  competency.  Because  of  the  large  scope  of  what  patient  safety  entails,  a  suggestion  was 
proposed  that  non-simulation  experts,  e.g.  economists  with  human  factors  engineering  experience, 
perform  the  assessment  of  simulation  usage  for  patient  safety. 

Conclusions 

The  increasing  multidisciplinary  membership  of  Society  for  Simulation  in  Healthcare  has  generated  a 
demand  for  consistent  and  validated  curricula  and  assessment  tools  for  the  expanding  applications  using 
simulation,  as  evident  by  the  questions  posed  on  the  SSH  listserv.  The  mission  of  SSH  is  to  facilitate 
excellent  multi-specialty  healthcare  education,  practice,  and  research  through  simulation.  Therefore,  to 
better  represent  our  multidisciplinary  simulation  society,  we  need  to  consolidate  the  needs  and  desires  of 
different  agencies  and  healthcare  specialties  in  order  to  establish  and  implement  a  unified  set  of 
educational  standards  for  simulation  applications. 

This  second  Simulation  Education  Summit  is  a  testament  to  the  wonders  of  collaborative  efforts.  As  a 
result  of  bringing  together  the  key  stakeholders  in  simulation  education,  this  preliminary  outline  of 
considerations  for  simulation-based  training  and  assessment  standards  has  been  drafted.  There  is  no 
doubt  that  much  more  work  is  needed  before  a  more  comprehensive  set  of  standards  or  a  guidelines 
document  can  be  distributed.  However,  this  report  serves  as  a  starting  point  to  address  the  same 
questions  that  every  organization  faces.  The  fact  that  nearly  all  Summit  participants  indicated  that  we 
should  continue  to  meet  in  this  collaborative  forum  and  not  wait  a  year  to  reconvene  is  an  indication  of  the 
strong  desire  to  work  together  toward  a  common  goal.  At  the  conclusion  of  the  meeting,  the  group  agreed 
on  an  action  plan. 

Action  Plan  and  Recommendations 

•  The  Society  for  Simulation  in  Healthcare  will  form  a  coalition  of  societies  interested  in  simulation 
to  serve  as  the  “think  tank”  in  resolving  simulation-based  issues. 

•  After  further  discussion  and  consensus,  SSH  will  convene  an  ad  hoc  development  group  to 
create  the  standards  documents  and  guidelines  for  review  and  dissemination. 

•  The  next  meeting  should  include  definitions  of  standards  and  contextual  framing  to  facilitate 
better  discussions. 

•  SSH  should  act  as  a  clearinghouse  to  survey  practices  within  organizations. 

•  Along  with  input  from  specialty  societies  and  accrediting  agencies,  SSH  will  continue  to  lead 
these  efforts  in  consolidating  common  simulation  needs  and  issues. 


12 


Acknowledgements 

The  authors  acknowledge  that  this  paper  is  a  compilation  of  ideas  from  the  Summit  participants  and 
therefore  credit  is  given  to  all  who  participated.  We  thank  Beverlee  Anderson  for  her  organization  and 
leadership,  Sandra  Davis,  RN,  PhD  for  her  review  of  the  paper,  William  Dunn,  MD  for  his  references  on 
validated  simulation  instruments,  and  Jay  Vogt  for  his  time  in  planning,  preparations,  and  collation  of  the 
meeting  notes. 

References 

17.  Sinz  L.  2006  Simulation  Summit.  Simul  Healthcare  2007;  2(1  ):33-39. 

18.  Swanstrom  LL  et  al.  Beta  test  results  of  a  new  system  in  laparoscopic  surgery.  J  Am  Coll  Surg  2006; 

202(1  ):62-69 

19.  Fried  GM  et  al.  Proving  the  value  of  simulation  in  laparoscopic  surgery.  Ann  Surg  2004;  240(3):51 8-528 

20.  Gallagher  AG  et  al.  Discriminative  validity  of  the  Minimally  Invasive  Surgical  Trainer  in  Virtual  Reality  (MIST- 
VR)  using  criteria  levels  based  on  expert  performance.  Surg  Endosc  2004;18(4):660-665 

21 .  Seymour  NE  et  al.  Virtual  reality  training  improves  operating  room  performance:  results  of  a  randomized, 
double-blinded  study.  Ann  Surg  2002;  236  (4):458-464 

22.  Sedlack  RE,  Kolars  JC.  Computer  simulator  training  enhances  the  competency  of  gastroenterology  fellows 
at  colonoscopy:  results  of  a  pilot  study.  Am  J  Gastroenterol.  2004  Jan;99(1  ):33-7 

23.  Ritter  EM,  McClusky  DA  3rd,  Lederman  AB,  Gallagher  AG,  Smith  CD.  Objective  psychomotor  skills 
assessment  of  experienced  and  novice  flexible  endoscopists  with  a  virtual  reality  simulator.  J  Gastrointest 
Surg.  2003  Nov;7(7):871-7;  discussion  877-8. 

24.  Sedlack  RE,  Kolars  JC.  Validation  of  a  computer-based  colonoscopy  simulator.  Gastrointest  Endosc.  2003 
Feb;57(2):214-8. 

25.  Blum  MG,  Powers  TW,  Sundaresan  S.  Ann  Thorac  Surg.  Bronchoscopy  simulator  effectively  prepares  junior 
residents  to  competently  perform  basic  clinical  bronchoscopy.  2004  Jul;78(1):287-91;  discussion  287-91. 

26.  Colt  HG,  Crawford  SW,  Galbraith  O  3rd.  Virtual  reality  bronchoscopy  simulation:  a  revolution  in  procedural 
training.  Chest.  2001  Oct;120(4):1333-9. 

27.  Mayo  PH,  Hackney  JE,  Mueck  JT,  Ribaudo  V,  Schneider  RF.  Achieving  house  staff  competence  in 
emergency  airway  management:  results  of  a  teaching  program  using  a  computerized  patient  simulator.  Crit 
Care  Med.  2004  Dec;32(12):2422-7. 

28.  Patel  AD,  Gallagher  AG,  Nicholson  WJ,  Cates  CU.  Learning  curves  and  reliability  measures  for  virtual  reality 
simulation  in  the  performance  assessment  of  carotid  angiography.  J  Am  Coll  Cardiol.  2006  May 

2;47(9):1 796-802.  JACC  47:1796-1802,  2006 

29.  Edmond  CV  Jr.  Impact  of  the  endoscopic  sinus  surgical  simulator  on  operating  room  performance. 
Laryngoscope.  2002  Jul;1 12(7  Pt  1  ):1 148-58.  Laryngoscope  1 12:1 148-1 158,  2002 

30.  CPM  van  der  Vleuten  and  David  Swanson:  Assessment  of  Clinical  Skills  with  Standardized  Patients:  State 
of  the  Art.  Teach  Learn  Med  1990;  2(2):58-76 

31.  Jerry  Colliverand  Reed  Williams:  Technical  Issues:  Test  Application.  Acad  Med  1993;  68:454-460. 

32.  Fletcher,  G.,  Flin,  R.,  McGeorge,  P.,  Glavin,  R.,  Maran,  N.,  &  Patey,  R.  (2003).  Anaesthetists'  Non-Technical 
Skills  (ANTS):  Evaluation  of  a  behavioural  marker  system.  British  Journal  of  Anaesthesia,  90  (5),  580  -  588. 
Also  see  http://www.abdn.ac.uk/iprc/ants  for  more  information. 


Appendix  A:  List  of  Organizations  Represented  at  the  2007  Summit 

1.  Association  of  American  Medical  Colleges 

2.  American  Academy  of  Ophthalmology 

3.  American  Academy  of  Orthopedic  Surgeons 

4.  American  Association  for  Clinical  Anatomists 

5.  American  Association  of  Nurse  Anesthetists 

6.  American  Board  of  Medical  Specialties 

7.  American  Board  of  Surgery 

8.  American  College  of  Cardiology 

9.  American  College  of  Chest  Physicians 

10.  American  College  of  Surgeons 

11.  American  Dental  Society  of  Anesthesiology 

12.  American  Heart  Association 

13.  American  Heart  Association,  National  Center 

14.  American  Hospital  Association 

15.  American  Academy  of  Pediatric  Dentistry 

16.  American  Association  of  Oral  &  Maxillofacial  Surgeons 

17.  American  Medical  Foundation  for  Peer  Review  and  Education 

18.  American  Society  for  Health  Risk  Management  AHA 

19.  American  Society  of  Anesthesiologists 


13 


20.  Anesthesia  Patient  Safety  Foundation 

21.  Association  of  Standardized  Patient  Educators 

22.  California  Institute  of  Nursing  &  Education 

23.  Canadian  Patient  Safety  Institute 

24.  Infusion  Nurses  Certification  Corporation 

25.  Institute  of  Medicine  of  the  National  Academies 

26.  International  Association  of  Medical  Science  Educators 

27.  International  Nursing  Association  for  Clinical  Simulation  &  Learning 

28.  Joint  Commission  Accreditation  of  Hospital  Organizations 

29.  National  Board  of  Osteopathic  Medical  Examiners 

30.  National  Council  of  State  Boards  of  Nursing 

31.  National  League  of  Nursing 

32.  National  Nursing  Staff  Development  Organization 

33.  Society  of  Neurological  Surgeons 

34.  Society  for  Academic  Emergency  Medicine 

35.  Society  for  Simulation  in  Healthcare 

36.  Telemedicine  and  Advanced  Technologies  Research  Center  (TATRC) 


14 


Appendix  B:  Simulation  Taxonomy 


•  Simulation:  A  technique  that  uses  a  situation  or  environment  created  to  allow  persons  to 
experience  a  representation  of  a  real  event  for  the  purpose  of  practice,  learning,  evaluation, 
testing,  or  to  gain  understanding  of  systems  or  human  actions. 

•  Simulation  program:  A  formal  workshop,  course,  class,  or  other  activity  that  uses  a 
substantial  component  of  simulation  as  a  technique. 

•  Simulation  fidelity:  The  physical,  contextual,  and  emotional  realism  that  allows  persons  to 
experience  a  simulation  as  if  they  were  operating  in  an  actual  healthcare  activity. 

•  Formative  evaluation:  A  process  for  determining  the  competence  of  a  person  engaged  in  a 
healthcare  activity  for  the  purpose  of  providing  constructive  feedback  for  that  person  to 
improve. 

•  Summative  evaluation:  A  process  for  determining  the  competence  of  a  person  engaged  in  a 
healthcare  activity  for  the  purpose  of  certifying  with  reasonable  certainty  that  they  are  able  to 
perform  that  activity  in  practice. 

•  Simulation  validity:  The  quality  of  a  simulation  or  simulation  program  that  demonstrates  that 
the  relationship  between  the  process  and  its  intended  purpose  is  specific,  sensitive,  reliable, 
and  reproducible. 

•  Simulation  guideline:  A  recommendation  of  the  qualities  for  simulation  fidelity,  simulation 
validity,  simulation  program,  or  for  formative  or  summative  evaluation. 

•  Simulation  standard:  A  statement  of  the  minimum  requirements  for  simulation  fidelity, 
simulation  validity,  simulation  program,  or  for  formative  or  summative  evaluation. 

•  Certification  of  simulation:  The  process  of  approving  that  a  simulation  or  simulation 
program  meets  certain  published  standards. 
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Table  1.  A  Preliminary  List  of  Validated  Simulators  and  Evaluation  Tools 


Simulation/  evaluation  tool 

Reference  # 

Laparoscopic  trainer/  virtual  cholecystectomy 

2-5 

Gastrointestinal  endoscopy 

6-7 

Bronchoscopy 

9-10 

Emergency  airway  management 

11 

Carotid  angiography 

12 

Endoscopic  sinus  surgery 

13 

Standardized  patients 

14-15 

Anesthesia  Non-technical  Skills  (ANTS) 

16 

Note:  Simulation  is  one  technique  of  training  and  assessment  in  a  person’s  portfolio.  It  is  assumed  that  a 
range  of  different  educational  techniques  (didactic,  apprentice,  workshop,  etc.)  and  measurement  tools 
(e.g.  written  test,  oral  exams,  case  logs,  subjective  clinical  evaluations,  etc.)  will  be  needed. 
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